Altmetrics (Chapter from Beyond Bibliometrics: Harnessing Multidimensional Indicators of Scholarly Impact)

or to skim the text of the article quickly, while academics are more likely to download and print the paper. Further research investigating the ratio between HTML views and PDF downloads could uncover interesting findings about how the public interacts with the open access (OA) research literature. Scholars In addition to tracking scholarly impacts on traditionally invisible audiences, altmetrics hold potential for tracking previously hidden scholarly impacts. Faculty of 1000 Faculty of 1000 (F1000) is a service publishing reviews of important articles, as adjudged by a core “faculty” of selected scholars. Wets, Weedon, and Velterop (2003) argue that F1000 is valuable because it assesses impact at the article level, and adds a human level assessment that statistical indicators lack. Others disagree (Nature Neuroscience, 2005), pointing to a very strong correlation (r = 0.93) between F1000 score and Journal Impact Factor. This said, the service has clearly demonstrated some value, as over two thirds of the world’s top research institutions pay the annual subscription fee to use F1000 (Wets et al., 2003). Moreover, F1000 has been to shown to spot valuable articles which “sole reliance on bibliometric indicators would have led [researchers] to miss” (Allen, Jones, Dolby, Lynn, & Walport, 2009, p. 1). In the PLoS dataset, F1000 recommendations were not closely associated with citation or other altmetrics counts, and formed their own factor in factor analysis, suggesting they track a relatively distinct sort of impact. Conversation (scholarly blogging) In this context, “scholarly blogging” is distinguished from its popular counterpart by the expertise and qualifications of the blogger. While a useful distinction, this is inevitably an imprecise one. One approach has been to limit the investigation to science-only aggregators like ResearchBlogging (Groth & Gurney, 2010; Shema & Bar-Ilan, 2011). Academic blogging has grown steadily in visibility; academics have blogged their dissertations (Efimova, 2009), and the ranks of academic bloggers contain several Fields Medalists, Nobel laureates, and other eminent scholars (Nielsen, 2009). Economist and Nobel laureate Paul Krugman (Krugman, 2012), himself a blogger, argues that blogs are replacing the working-paper culture that has in turn already replaced economics journals as distribution tools. Given its importance, there have been surprisingly few altmetrics studies of scholarly blogging. Extant research, however, has shown that blogging shares many of the characteristics of more formal communication, including a long-tail distribution of cited articles (Groth & Gurney, 2010; Shema & Bar-Ilan, 2011). Although science bloggers can write anonymously, most blog under their real names (Shema & Bar-Ilan, 2011). Conversation (Twitter) Scholars on Twitter use the service to support different activities, including teaching (Dunlap & Lowenthal, 2009; Junco, Heiberger, & Loken, 2011), participating in conferences (Junco et al., 2011; Letierce et al., 2010; Ross et al., 2011), citing scholarly articles (Priem & Costello, 2010; Weller, Dröge, & Puschmann, 2011), and engaging in informal communication (Ross et al., 2011; Zhao & Rosson, 2009). Citations from Twitter are a particularly interesting data source, since they capture the sort of informal discussion that accompanies early important work. There is, encouragingly, evidence that Tweeting scholars take citations from Twitter seriously, both in creating and reading them (Priem & Costello, 2010). The number of scholars on Twitter is growing steadily, as shown in Figure 1. The same study found that, in a sample of around 10,000 Ph.D. students and faculty members at five representative universities, one 1 in 40 scholars had an active Twitter account. Although some have suggested that Twitter is only used by younger scholars, rank was not found to significantly associate with Twitter use, and in fact faculty members’ tweets were twice as likely to discuss their and others’ scholarly work. Conversation (article commenting) Following the lead of blogs and other social media platforms, many journals have added article-level commenting to their online platforms in the middle of the last decade. In theory, the discussion taking place in these threads is another valuable lens into the early impacts of scientific ideas. In practice, however, many commenting systems are virtual ghost towns. In a sample of top medical journals, fully half had commenting systems laying idle, completely unused by anyone (Schriger, Chehrazi, Merchant, & Altman, 2011). However, commenting was far from universally unsuccessful; several journals had comments on 50-76% of their articles. In a sample from the British Medical Journal, articles had, on average, nearly five comments each (Gotzsche, Delamothe, Godlee, & Lundh, 2010). Additionally, many articles may accumulate comments in other environments; the growing number of external comment sites allows users to post comments on journal articles published elsewhere. These have tended to appear and disappear quickly over the last few years. Neylon (2010) argues that online article commenting is thriving, particularly for controversial papers, but that "...people are much more comfortable commenting in their own spaces” (para. 5), like their blogs and on Twitter. Reference managers Reference managers like Mendeley and CiteULike are very useful sources of altmetrics data and are currently among the most studied. Although scholars have used electronic reference managers for some time, this latest generation offers scientometricians the chance to query their datasets, offering a compelling glimpse into scholars’ libraries. It is worth summarizing three main points, though. First, the most important social reference managers are CiteULike and Mendeley. Another popular reference manager, Zotero, has received less study (but see Lucas, 2008). Papers and ReadCube are newer, smaller reference managers; Connotea and 2Collab both dealt poorly with spam; the latter has closed, and the former may follow. Second, the usage base of social reference managers—particularly Mendeley—is large and growing rapidly. Mendeley’s coverage, in particular, rivals that of commercial databases like Scopus and Web of Science (WoS) (Bar-Ilan et al., 2012; Haustein & Siebenlist, 2011; Li et al., 2011; Priem et al., 2012). Finally, inclusion in reference managers correlates to citation more strongly than most other altmetrics. Working with various datasets, researchers have reported correlations of .46 (Bar-Ilan, 2012), .56 (Li et al., 2011), and .5 (Priem et al., 2012) between inclusion in users’ Mendeley libraries, and WoS citations. This closer relationship is likely because of the importance of reference managers in the citation workflow. However, the lack of perfect or even strong correlation suggests that this altmetric, too, captures influence not reflected in the citation record. There has been particular interest in using social bookmarking for recommendations (Bogers & van den Bosch, 2008; Jiang, He, & Ni, 2011). pdf downloads As discussed earlier, most research on downloads today does not distinguish between HTML views in PDF downloads. However there is a substantial and growing body of research investigating article downloads, and their relation to later citation. Several researchers have found that downloads predict or correlate with later citation (Perneger, 2004; Brody et al., 2006). The MESUR project is the largest of these studies to date, and used linked usage events to create a novel map of the connections between disciplines, as well as analyses of potential metrics using download and citation data in novel ways (Bollen, et al., 2009). Shuai, Pepe, and Bollen (2012) show that downloads and Twitter citations interact, with Twitter likely driving traffic to new papers, and also reflecting reader interest. Uses, limitations and future research Uses Several uses of altmetrics have been proposed, which aim to capitalize on their speed, breadth, and diversity, including use in evaluation, analysis, and prediction. Evaluation The breadth of altmetrics could support more holistic evaluation efforts; a range of altmetrics may help solve the reliability problems of individual measures by triangulating scores from easily-accessible “converging partial indicators” (Martin & Irvine, 1983, p. 1). Altmetrics could also support the evaluation of increasingly important, non-traditional scholarly products like datasets and software, which are currently underrepresented in the citation record (Howison & Herbsleb, 2011; Sieber & Trumbo, 1995). Research that impacts wider audiences could also be better rewarded; Neylon (2012) relates a compelling example of how tweets reveal clinical use of a research paper—use that would otherwise go undiscovered and unrewarded. The speed of altmetrics could also be useful in evaluation, particularly for younger scholars whose research has not yet accumulated many citations. Most importantly, altmetrics could help open a window on scholars’ “scientific ‘street cred’” (Cronin, 2001, p. 6), helping reward researchers whose subtle influences—in conversations, teaching, methods expertise, and so on— influence their colleagues without perturbing the citation record. Of course, potential evaluators must be strongly cautioned that while uncritical application of any metric is dangerous, this is doubly so with altmetrics, whose research base is not yet adequate to support high-stakes decisions.